Displaying answers in accordance with answer classifications

ABSTRACT

A question is received in a computerized answering system via an input device. A plurality of classes corresponding to the question are retrieved from a memory device. One or more answers to the question are generated by a processor. Each answer is associated with a confidence value. The confidence values associated with the answers are compared with thresholds associated with the plurality of classes. The answers are classified by the processor into the plurality of classes based on the comparing. The answers are presented via a display device in accordance with the classification of the answers. A user response to at least one of the answers is detected. At least one of the plurality of classes is adjusted based on the user response.

BACKGROUND

Embodiments of the inventive subject matter generally relate to thefield of computer systems, and, more particularly, to the use of userbehavior data to facilitate in the operation of a question answeringsystem.

SUMMARY

Embodiments generally include a method that includes receiving, via aninput device, a question in the computerized answering system. Themethod further includes retrieving, from a memory device, a plurality ofclasses corresponding to the question. The method further includesgenerating, by a processor, one or more answers to the question. Eachanswer is associated with a confidence value. The method furtherincludes comparing the confidence values associated with the answerswith thresholds associated with the plurality of classes. The methodfurther includes classifying, by the processor, the answers into theplurality of classes based on the comparing. The method further includespresenting, via a display device, the answers in accordance with theclassification of the answers. The method further includes detecting auser response to at least one of the answers. The method furtherincludes adjusting at least one of the plurality of classes based on theuser response.

BRIEF DESCRIPTION OF THE DRAWINGS

The present embodiments may be better understood, and numerous objects,features, and advantages made apparent to those skilled in the art byreferencing the accompanying drawings.

FIG. 1 is a conceptual diagram illustrating a QA system that analyzesuser behavior to determine adjustments for answer classes in a QApipeline.

FIG. 2 depicts a flow diagram illustrating example operations foradjusting answer classes based on user behavior.

FIG. 3 is a conceptual diagram illustrating a QA system that analyzesuser behavior to determine a user profile.

FIG. 4 depicts a flow diagram illustrating example operations forselecting a user profile based on user behavior.

FIG. 5 depicts an example computer system with an interface modificationunit.

DESCRIPTION OF EMBODIMENT(S)

The description that follows includes exemplary systems, methods,techniques, instruction sequences and computer program products thatembody techniques of the present inventive subject matter. However, itis understood that the described embodiments may be practiced withoutthese specific details. For instance, although examples refer toclassifying answers using associated answer confidence values, answersmay be classified using other criteria such as a user rating for ananswer, whether an answer is known to be good, or the amount of evidencefor an answer. In other instances, well-known instruction instances,protocols, structures and techniques have not been shown in detail inorder not to obfuscate the description.

A question answering (“QA”) system allows a user to submit a questionfor answering. The QA system generally returns a number of possibleanswers that are displayed to a user. The manner in which the answersare displayed is typically predetermined by a QA system interface. Forexample, the interface may be configured to display only the top fiveanswers based on associated answer confidence values determined by theQA system or may be configured to display all returned answers. Apredetermined, static interface limits the ability of the QA system toaccommodate specific users or applications. For instance, in someapplications of a QA system, it may be suitable to display only the topanswer or to also display supporting evidence supplied by the QA systemalong with the returned answer. For such situations, the QA system canutilize user behavior data to determine how to adjust the interface. Asan example, in one application, users may consistently rank answers withconfidence values above 80 on a scale of 0-100 as being correct oruseful answers. The QA system could use this user behavior data tomodify the interface for that application to display only a single topanswer when there is a returned answer with a confidence value greaterthan 80. In another example, a specific user may consistently click toview the supporting evidence for a returned answer. The QA system couldmodify the interface to fit the specific user's behavior and alwaysdisplay supporting evidence for a returned answer automatically. In thisway, the QA system can dynamically configure the interface in responseto specific user behavior.

FIG. 1 is a conceptual diagram illustrating a QA system that analyzesuser behavior to determine adjustments for answer classes in a QApipeline. FIG. 1 depicts a QA system 100 that includes a QA pipeline101, a user behavior monitor 102, an answer class adjustment unit 103,an answer generator 105, and a QA interface module 106. The QA pipeline101 and the answer class adjustment unit 103 communicate with an answerclass database 104. QA interface module 106 outputs a sample userinterface 107. FIG. 1 depicts a series of stages A-G. These stagesillustrate example operations and should not be used to limit scope ofany claims.

At stage A, the QA pipeline 101 (“pipeline 101”) receives a questionfrom the QA interface module 106 (“interface module 106”). The interfacemodule 106 receives the question through the user interface 107. Theuser interface 107 may be displayed within an application or web browseron a computer, mobile phone, or other client device and allows a user tosubmit a question by inputting text, selecting a question from a list,using voice recognition, or other input method. The interface module 106may be located remotely on a server or locally on a client device. Theinterface module 106 controls how data is displayed in the userinterface 107 and manages input to the QA system 100 that is receivedfrom a number of user interfaces 107. The pipeline 101 serves as acentral hub that connects modules of the QA system 100. The pipeline 101may collect and organize data, such as answers, confidence values, andthresholds, from different modules to send to the interface module 106.The QA system 100 might contain multiple QA pipelines that eachcorrespond to a specific subject matter or corpus of documents. Forexample, if the pipeline 101 is designed for use in the medicalindustry, the pipeline 101 may direct questions to a corpus of documentsrelating to medicine. Additionally, since accuracy is important in amedical setting, the pipeline 101 may be configured to only returnanswers with a high confidence value. After receiving the question, thepipeline 101 sends the question to the answer generator 105.

At stage B, the answer generator 105 returns answers to the pipeline101. The answer generator 105 generates a set of possible answers andcorresponding confidence values to the question and then sends them tothe QA pipeline 101. Answers may be generated by using informationretrieval techniques. For example, the answer generator 105 may performnatural language processing on the question and then search a corpus ofdocuments that has been ingested into the QA system 100. The answergenerator 105 may identify one or more answers within the corpus for agiven question. Each answer is associated with an answer confidencescore that is typically within a scale of 0-100. Once determined theanswers and confidence values are sent to the QA pipeline 101.

At stage C, the pipeline 101 retrieves thresholds from the answer classdatabase 104 (“class database 104”). The class database 104 contains anumber of thresholds for questions with different attributes. Thepipeline 101 identifies question attributes by performing naturallanguage processing on the question. Question attributes include thequestion classification, the lexical answer type (“LAT”), and the focus.A question classification may indicate the answer type of a questionsuch as factoid, procedural, etc. The LAT may be a word in the questionthat indicates the type of answer being asked for, and the focus of thequestion is a reference to the answer. For example, the question “whowas the first man on the moon?” would have a question classification offactoid, have a LAT of “man,” and have a focus of “who.” Additionalquestion attributes may include qualifiers, quantifiers, cardinals,ordinals, etc. Different classes may be determined for each questionwith a unique attribute set. For example, as depicted in FIG. 1, theclass database 104 contains classes for questions with a questionclassification factoid and a question classification procedural. Theclass database 104 may also contain classes for more specific questionattributes. For example, classes may exist for a question with theattribute set of a question classification factoid, a LAT “person,” anda focus “where.” After identifying the question attributes, the pipeline101 can retrieve the corresponding classes from the class database 104using the attribute set.

At stage D, the pipeline 101 organizes the answers received from theanswer generator 105. The pipeline 101 organizes the answers based onthe thresholds of classes retrieved from the class database 104 at stageC. Classes have thresholds that indicate which answers should beclassified in the class based on the answers' confidence values.Additionally, the class contains rules for how to display answers in theclass. For example, a class may have a threshold of “if <50” and a ruleof “do not display answer,” which would cause the pipeline 101 toclassify answers with confidence values of less than 50 into the classand cause the interface module 106 to prevent the answer in the classfrom being displayed in the interface 107. The pipeline 101 comparesanswer confidence values to the class thresholds to classify the answersinto classes. The pipeline 101 may also be configured to classifyanswers based on other criteria such as a user rating for an answer.Once the answers have been classified, the answers are sent in theirclasses to the interface module 106.

At stage E, the interface module 106 outputs the answers to be displayedon the user interface 107. The interface module 106 determines how theanswers should be displayed based on the class and adjusts the userinterface 107 accordingly. For example, as depicted in FIG. 1, thequestion “how do I charge my device?” has a question classification ofprocedural, so the classes that are retrieved from the class database104 are “if >90, display answer,” “if 60-90, click to view answer,” and“if <60, do not display answer.” Based on the classes, the interfacemodule 106 modifies the user interface 107 so that the answers in thefirst class are displayed, the answers in the second class are availableto view if clicked, and the answers in the third class are notdisplayed. Once a new question is submitted, the interface module 106will again modify the user interface 107 to display answers according tonew classes.

At stage F, the user behavior monitor 102 (“monitor 102”) receives userbehavior data through the interface module 106. The user interface 107allows a user to rank an answer and submit that ranking to the QA system100 through the interface module 106. For example, as depicted in FIG.1, a user may click below an answer to rate the answer as “good” or“bad.” More complex interfaces may also allow a user to rank an answeron a scale or submit comments about an answer. The monitor 102 tracksuser responses to answers and determines whether a certain answer isreceiving a positive or negative user response. As depicted in FIG. 1,the monitor 102 tracks whether the ranking for a specific answer is“good” or “bad” to determine a positive or negative response. If noranking is submitted, the monitor 102 may also determine, for example,that the response to “Answer 1” and “Answer 2” is negative if a userclicks to view “Answer 3” or “Answer 4.” In some embodiments, themonitor 102 may perform language processing on user comments for ananswer to determine a response. Once determined, the user response datais sent to the answer class adjustment unit 103 (“adjustment unit 103”).

At stage G, the adjustment unit 103 determines whether to adjust theclasses. The adjustment unit 103 analyzes the user response datareceived from the monitor 102 to determine whether an adjustment isnecessary. An adjustment may be necessary if an answer within a certainclass is receiving one or more positive or negative user responses. Forexample, as depicted in FIG. 1, “Answer 3” has been placed according toits confidence value into the class for question classificationprocedural of “if 60-90, click to view answer.” If “Answer 3” hasreceived a number of positive responses, then the adjustment unit 103will determine that “Answer 3” should be displayed automatically andwill adjust the class threshold of “if >90, display answer” to include“Answer 3.” If, for example, “Answer 3” has a confidence value of 85,the class threshold may be adjusted to “if >=85, display answer.”Similarly, if “Answer 2” is receiving a number of negative responses,the adjustment unit 103 may adjust the class threshold for questionclassification procedural of “if >90, display answer” to exclude answerswith confidence values equal to or below the confidence value of “Answer2.” If determined that a class adjustment is needed, the adjustment unit103 will adjust the classes and store the adjusted classes in the classdatabase 104 for future use by the pipeline 101. The adjustment unit 103may continue to adjust the classes as more questions are asked andadditional user response data is determined.

FIG. 2 depicts a flow diagram illustrating example operations foradjusting answer classes based on user behavior.

At block 200, a question is received by a QA pipeline. Once a query hasbeen received, control then flows to block 202.

At block 202, question attributes are identified. As described above,examples of question attributes include the question classification, theLAT, and the focus. The question attributes are identified using naturallanguage processing, which may be performed by the QA pipeline or by aseparate module of a QA system. Once the question attributes areidentified, control then flows to block 204.

At block 204, classes corresponding to the question attributes and theQA pipeline are retrieved. Classes may be retrieved from a databasewhere they are stored or they may reside locally in storage. Each QApipeline has a different set of classes unique to the pipeline.Similarly, each set of question attributes may have its own uniqueclasses. Classes may be retrieved by identifying the set of classes thatcorrespond to the QA pipeline and the question attributes. Also, classesmay be stored in a database and be retrieved by querying the database.Once the classes are retrieved, control then flows to block 206.

At block 206, answer confidence values are compared to the classthresholds. As mentioned in the FIG. 1 description, classes havethresholds for classifying answers by their answer confidence valuesinto a class. The answers may be classified according to the classthresholds by the QA pipeline or by another module of a QA system. Oncethe answers are classified, control then flows to block 208.

At block 208, answers are displayed according to the classification ofthe answers. Classes have rules that dictate how answers in the classshould be displayed. As mentioned above in the FIG. 1 description, aclass rule may be to display answers in the class, to make answersavailable to view, or to not display answers. The answers may bedisplayed through an interface that is part of a QA system, or theanswers may be sent to an external client interface, such as a web pageor mobile device. Once the answers have been displayed, control thenflows to block 210.

At block 210, user behavior related to an answer is monitored. Theinterface through which the answers are displayed allows a user to rate,rank, or give feedback for the answers. By monitoring the behavior, a QAsystem can determine if a response to a returned answer is positive ornegative. Additionally, by performing language processing on userfeedback, a QA system can determine if an answer is useful in somemanner, so even if an answer receives a negative rating, the feedbackmay indicate that the answer should still be displayed. After userbehavior has been monitored, control then flows to block 212.

At block 212, it is determined whether the user behavior indicates aclass adjustment. The user responses determined at block 210 indicatewhether to adjust a class. For example, if a positive response to ananswer that is not displayed is indicated, the class may be adjusted toautomatically display answers with similar confidence values in thefuture. Conversely, if a negative response to an answer that is alreadynot displayed automatically is indicated, the class may not be adjusted.If the user behavior does not indicate a class adjustment, the processends. If the user behavior does indicate a class adjustment, controlthen flows to block 214.

At block 214, the class is adjusted in accordance with the userbehavior. A class may be adjusted by changing the class threshold. Forexample, a class threshold may be “if >80, display answer,” but if theuser behavior indicates that answers with confidence values of 70 orgreater are receiving a positive user response, then the class thresholdmay be adjusted to “if >70, display answer.” A class may also beadjusted by changing the class rule. For example, a class may be“if >75, display answer.” If a positive response is indicated only aftera user has viewed evidence for an answer, the class rule of “displayanswer” may be changed to “display answer and answer evidence.” Theadjusted class may be stored wherever the original class is located, adatabase, local storage, etc. The adjusted class will replace theexisting class so that the adjusted class will be selected in the futurefor questions with a same set of attributes. Once the class has beenadjusted, the process ends.

FIG. 3 is a conceptual diagram illustrating a QA system that analyzesuser behavior to determine a user profile. FIG. 3 depicts a QA system300 that includes a QA pipeline 301, a user behavior monitor 302, a userprofile selection unit 303, an answer generator 305, and a QA interfacemodule 306. The user profile selection unit 103 communicates with a userprofile database 304. QA interface module 306 outputs a sample userinterface 307. FIG. 3 depicts a series of stages A-D. These stagesillustrate example operations and should not be used to limit scope ofany claims.

At stage A, the QA pipeline 301 (“pipeline 301”) receives a question andreturns possible answers. The pipeline 301 receives a question throughthe QA interface module 306 (“interface module 306”). The pipeline 301sends the question to the answer generator 305, which determines answersand returns a set of possible answers to the pipeline 301. The pipeline301 then returns the answers to the interface module 306. The interfacemodule 306 sends the answers to be displayed by user interface 307.

At stage B, the user behavior monitor 302 (“monitor 302”) receives userbehavior data through the interface module 306. The monitor 302 tracksuser activity and feedback conducted within the user interface 307. Forexample, the monitor 302 tracks when a user clicks to view answers thatare not displayed or clicks to view the evidence for an answer.Additionally, in more complex interfaces, the monitor 307 may also trackuser comments and user ratings for answers. The monitor 302 tracks auser throughout a session with the QA system 300 and provides the userbehavior data to the user profile selection unit 303 (“selection unit303”).

At stage C, the selection unit 303 selects a user profile based on theuser behavior data. The selection unit 303 analyzes the user behaviordata to identify patterns that indicate a specific user profile. Forexample, the selection unit 303 may determine that a user consistentlyviews only the top answer and the evidence for the top answer. Based onthis behavior pattern, the selection unit 304 retrieves an appropriateuser profile (e.g., “User Profile 2”) from the user profile database304. “User Profile 2” provides that only the top answer be displayed,answer evidence for the top answer be displayed, and other answers behidden until clicked. After retrieving the user profile from the userprofile database 304, the selection unit 303 sends the user profile tothe interface module 306.

At stage D, the interface module 306 updates the user interface 307based on the user profile. The interface module 306 may by default use astandard profile for all users, so by updating the user profile, theuser interface 307 becomes tailored to the current user's behaviorpattern. Upon the next question submitted by the current user, theinterface module 306 will cause the returned answers to be displayed bythe user interface 307 according to the selected user profile. The userinterface 307 for other users will not be affected by the selection of auser profile for the current user. However, a user profile specific tothe pipeline 301 may be selected, which would affect all users of thespecific pipeline 301.

FIG. 4 depicts a flow diagram illustrating example operations forselecting a user profile based on user behavior.

At block 400, a user session is initiated. A user session is a period inwhich a specific user interacts with a QA system. A session may be aslong as a user is continuously interacting with the QA system, may be aspecific length of time, or may span each time a specific user interactswith the QA system. Once a user session is initiated, control then flowsto block 402.

At block 402, user behavior during the session is monitored. Userbehavior may include whether a user opts to view non-displayed answers,ignores displayed answers, clicks to view answer evidence, or gives apositive or negative response to answers of certain confidence values.The monitored behavior is collected into user behavior data. Once theuser behavior data is determined, control then flows to block 404.

At block 404, a behavior pattern for the user session is determined. Abehavior pattern is determined as described above in the FIG. 3description. After the user behavior pattern is determined, control thenflows to block 406.

At block 406, it is determined whether the user behavior pattern fits anexisting user profile. Existing user profiles may be stored in a remotedatabase or may be stored locally. A user profile can be identified bycomparing attributes of the user behavior pattern with attributes of thestored user profiles. For example, the user behavior pattern mayindicate that a user consistently views the top five answers and theanswer evidence for those answers. An existing user profile may have theattributes of automatically displaying the top five answers and theanswer evidence for those answers. If the behavior pattern does not fitan existing user profile, control then flows to block 412. If thebehavior pattern does fit an existing user profile, control then flowsto block 408.

At block 408, the existing user profile based on the behavior pattern isretrieved. The existing user profile is selected and retrieved from itsstorage location. After the user profile is retrieved, control thenflows to block 410.

At block 410, answers are displayed according to the user profile. Theinterface which displays the answers is updated to display the answeraccording to the retrieved user profile. The interface displaying theanswers may be reconfigured in order to display answer according to theattributes of the user profile. Once the answers have been displayedaccording to the user profile, the process ends.

Control flowed to block 412 if, at block 406, it was determined that thebehavior pattern does not fit an existing user profile. At block 412, auser profile with attributes of the user behavior pattern is created.The new user profile may be stored in a remote or local database forfuture use by the QA system. After the user profile is created, controlthen flows to block 414.

At block 414, the created user profile is retrieved. The created userprofile is retrieved for use by the QA system. After the created userprofile is retrieved, control then flows to block 416.

At block 416, answers are displayed according to the created userprofile. The interface which displays the answers is updated to displaythe answer according to the created user profile. The interfacedisplaying the answers may be reconfigured in order to display answeraccording to the attributes of the created user profile. Once theanswers have been displayed according to the created user profile, theprocess ends.

As will be appreciated by one skilled in the art, aspects of the presentinventive subject matter may be embodied as a system, method or computerprogram product. Accordingly, aspects of the present inventive subjectmatter may take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and hardware aspects that mayall generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present inventive subject mattermay take the form of a computer program product embodied in one or morecomputer readable medium(s) having computer readable program codeembodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent inventive subject matter may be written in any combination ofone or more programming languages, including an object orientedprogramming language such as Java, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present inventive subject matter are described withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems) and computer program products according toembodiments of the inventive subject matter. It will be understood thateach block of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, can be implemented by computer program instructions. Thesecomputer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

FIG. 5 depicts an example computer system with an interface modificationunit. A computer system includes a processor unit 501 (possiblyincluding multiple processors, multiple cores, multiple nodes, and/orimplementing multi-threading, etc.). The computer system includes memory507. The memory 507 may be system memory (e.g., one or more of cache,SRAM, DRAM, zero capacitor RAM, Twin Transistor RAM, eDRAM, EDO RAM, DDRRAM, EEPROM, NRAM, RRAM, SONOS, PRAM, etc.) or any one or more of theabove already described possible realizations of machine-readable media.The computer system also includes a bus 503 (e.g., PCI, ISA,PCI-Express, HyperTransport®, InfiniBand®, NuBus, etc.), a networkinterface 505 (e.g., an ATM interface, an Ethernet interface, a FrameRelay interface, SONET interface, wireless interface, etc.), and astorage device(s) 509 (e.g., optical storage, magnetic storage, etc.).The interface modification unit 511 embodies functionality to modify aninterface based on answer classes and user profiles. The interfacemodification unit 511 may adjust an interface based on theclassification of answers into classes and associated class rules, basedon the selection of a user profile, or a combination of answerclassifications and user profiles. Any one of these functionalities maybe partially (or entirely) implemented in hardware and/or on theprocessing unit 501. For example, the functionality may be implementedwith an application specific integrated circuit, in logic implemented inthe processing unit 501, in a co-processor on a peripheral device orcard, etc. Further, realizations may include fewer or additionalcomponents not illustrated in FIG. 5 (e.g., video cards, audio cards,additional network interfaces, peripheral devices, etc.). The processorunit 501, the storage device(s) 509, and the network interface 505 arecoupled to the bus 503. Although illustrated as being coupled to the bus503, the memory 507 may be coupled to the processor unit 501.

While the embodiments are described with reference to variousimplementations and exploitations, it will be understood that theseembodiments are illustrative and that the scope of the inventive subjectmatter is not limited to them. In general, techniques for displayinganswers based on answer classes and user profiles as described hereinmay be implemented with facilities consistent with any hardware systemor hardware systems. Many variations, modifications, additions, andimprovements are possible.

Plural instances may be provided for components, operations orstructures described herein as a single instance. Finally, boundariesbetween various components, operations and data stores are somewhatarbitrary, and particular operations are illustrated in the context ofspecific illustrative configurations. Other allocations of functionalityare envisioned and may fall within the scope of the inventive subjectmatter. In general, structures and functionality presented as separatecomponents in the exemplary configurations may be implemented as acombined structure or component. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents. These and other variations, modifications, additions, andimprovements may fall within the scope of the inventive subject matter.

Use of the phrase “at least one of . . . or” should not be construed tobe exclusive. For instance, the phrase “X comprises at least one of A,B, or C” does not mean that X comprises only one of {A, B, C}; it doesnot mean that X comprises only one instance of each of {A, B, C}, evenif any one of {A, B, C} is a category or sub-category; and it does notmean that an additional element cannot be added to the non-exclusive set(i.e., X can comprise {A, B, Z}).

1. A method for providing answers in a computerized answering systemcomprising: receiving, via an input device, a question in thecomputerized answering system; retrieving, from a memory device, aplurality of classes corresponding to the question; generating, by aprocessor, one or more answers to the question, wherein each answer isassociated with a confidence value; comparing the confidence valuesassociated with the answers with thresholds associated with theplurality of classes; classifying, by the processor, the answers intothe plurality of classes based on the comparing; presenting, via adisplay device, the answers in accordance with the classification of theanswers; detecting a user response to at least one of the answers; andadjusting at least one of the plurality of classes based on the userresponse.
 2. The method of claim 1, wherein said adjusting at least oneof the plurality of classes comprises: determining the response to oneof the answers classified in one of the plurality of classes to be anegative response; and adjusting the threshold of the one of theplurality of classes to exclude answers having the same confidence valueas the one of the answers from being classified into the one of theplurality of classes.
 3. The method of claim 1, wherein each of theplurality of classes comprises: a threshold value operable to indicatewhich ones of a set of answers belong in a class based on comparingassociated answer confidence values with the threshold value; and anindication of how the answers in the class are to be presented via thedisplay device.
 4. The method of claim 3, wherein the indication of howthe answers are to be presented comprises at least one of: an indicationthat the answers be displayed; an indication that the answers beavailable to view; or an indication that the answers not be displayed.5. The method of claim 1, wherein the confidence values fall within arange, and wherein higher confidence values indicate increasedconfidence that the answers are correct.
 6. The method of claim 1further comprising: monitoring user interaction with the display deviceof the answering system; selecting a user profile corresponding to theuser interaction; and presenting answers to a second question inaccordance with the selected user profile.
 7. The method of claim 6,wherein the user profile comprises at least one of: an indication thatonly a specified number of answers be displayed; an indication that onlyanswers with a certain confidence value be displayed; or an indicationthat evidence for answers be displayed.
 8. A computer program productfor providing answers in a computerized answering system comprising: acomputer readable storage medium having program instructions embodiedtherewith, the program instructions comprising program instructions to,receive, via an input device, a question in the computerized answeringsystem; retrieve, from a memory device, a plurality of classescorresponding to the question; generate, by a processor, one or moreanswers to the question, wherein each answer is associated with aconfidence value; compare the confidence values associated with theanswers with thresholds associated with the plurality of classes;classify, by the processor, the answers into the plurality of classesbased on the comparing; present, via a display device, the answers inaccordance with the classification of the answers; detect a userresponse to at least one of the answers; and adjust at least one of theplurality of classes based on the user response.
 9. The computer programproduct of claim 8, wherein the program instructions to adjust at leastone of the plurality of classes comprises the program instructions to:determine the response to one of the answers classified in one of theplurality of classes to be a negative response; and adjust the thresholdof the one of the plurality of classes to exclude answers having thesame confidence value as the one of the answers from being classifiedinto the one of the plurality of classes.
 10. The computer programproduct of claim 8, wherein each of the plurality of classes comprises:a threshold value operable to indicate which ones of a set of answersbelong in a class based on comparing associated answer confidence valueswith the threshold value; and an indication of how the answers in theclass are to be presented via the display device.
 11. The computerprogram product of claim 10, wherein the indication of how the answersare to be presented comprises at least one of: an indication that theanswers be displayed; an indication that the answers be available toview; or an indication that the answers not be displayed.
 12. Thecomputer program product of claim 8, wherein the confidence values fallwithin a range, and wherein higher confidence values indicate increasedconfidence that the answers are correct.
 13. The computer programproduct of claim 8 further having program instructions to: monitor userinteraction with the display device of the answering system; select auser profile corresponding to the user interaction; and present answersto a second question in accordance with the selected user profile. 14.The computer program product of claim 13, wherein the user profilecomprises at least one of: an indication that only a specified number ofanswers be displayed; an indication that only answers with a certainconfidence value be displayed; or an indication that evidence foranswers be displayed.
 15. An apparatus comprising: a processor; an inputdevice; a memory device; a display device; and a computer readablestorage medium having program instructions embodied therewith, theprogram instructions executable by the processor to cause the apparatusto, receive, via the input device, a question; retrieve, from the memorydevice, a plurality of classes corresponding to the question; generate,by the processor, one or more answers to the question, wherein eachanswer is associated with a confidence value; compare the confidencevalues associated with the answers with thresholds associated with theplurality of classes; classify, by the processor, the answers into theplurality of classes based on the comparing; present, via the displaydevice, the answers in accordance with the classification of theanswers; detect a user response to at least one of the answers; andadjust at least one of the plurality of classes based on the userresponse.
 16. The apparatus of claim 15, wherein the programinstructions executable by the processor to cause the apparatus toadjust at least one of the plurality of classes comprises the programinstructions executable by the processor to cause the apparatus to:determine the response to one of the answers classified in one of theplurality of classes to be a negative response; and adjust the thresholdof the one of the plurality of classes to exclude answers having thesame confidence value as the one of the answers from being classifiedinto the one of the plurality of classes.
 17. The apparatus of claim 15,wherein each of the plurality of classes comprises: a threshold valueoperable to indicate which ones of a set of answers belong in a classbased on comparing associated answer confidence values with thethreshold value; and an indication of how the answers in the class areto be presented via the display device.
 18. The apparatus of claim 17,wherein the indication of how the answers are to be presented comprisesat least one of: an indication that the answers be displayed; anindication that the answers be available to view; or an indication thatthe answers not be displayed.
 19. The apparatus of claim 15 furtherhaving program instructions executable by the processor to cause theapparatus to: monitor user interaction with the display device of theanswering system; select a user profile corresponding to the userinteraction; and present answers to a second question in accordance withthe selected user profile.
 20. The apparatus of claim 19, wherein theuser profile comprises at least one of: an indication that only aspecified number of answers be displayed; an indication that onlyanswers with a certain confidence value be displayed; or an indicationthat evidence for answers be displayed.